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A variety  of  Virtual  Environment  (VE)  systems 
exist  today  and  there  are  as  many  terms  to  them. 
Synthetic  environments,  virtual  reality  (VR),  and 
VEs,  to  name  a few,  all  refer  to  a simulation  of 
some  operational  environment.  The  gamut  of  VEs 
runs  anywhere  from  desktop  displays  to  fully 
immersive  and  interactive  scenes.  What  these 
systems  share  is  the  need  to  capture,  process,  and 
present  data.  Presented  effectively,  the 
information  can  enable  a user  to  make  timely  and 
informed  decisions. 

This  paper  will  provide  some  examples  of  current 
VE  technology  and  present  a framework  for  VE 
systems.  The  paper  will  also  address  some  of  the 
future  directions  and  challenges  facing  the 
evolution  of  VE  technology. 

1.  Current  VE  Systems 

The  Office  of  Naval  Research  (ONR)  sponsors 
basic  research  and  advanced  technology 
demonstrations  in  the  area  of  VEs  for  training. 
The  ONR  VE  program  was  founded  partially 
upon  the  need  for  affordable,  compact, 
deployable,  and  reconfigurable  training  systems. 
Additionally,  the  VE  program  was  created  to 
address  reductions  in  manning  and  increasing 
training  requirements.  The  training  devices 
demonstrated  to  date  include  a pilot  trainer  for 
underwater  remotely  operated  vehicles  (ROV)  and 
a trainer  for  submarine  and  shiphandling. 

The  ROV  trainer  is  a desktop  VE  and  illustrates  at 
least  three  important  characteristics  of  VEs.  First, 
performance  tests  showed  that  trainees  displayed  a 
positive  training  transfer  from  the  desktop 
training  to  actual  operation.  As  will  be  discussed 


later,  the  utility  of  VE  systems  must  be 
demonstrated  in  actual  operative  results.  Second, 
the  system,  originally  built  upon  an  SGI  platform, 
has  been  transferred  to  a PC  version  - an 
important  step  in  developing  portable  systems  for 
ease  in  deployability.  Last,  the  desktop  system 
has  the  potential  for  use  in  other  types  of  remote 
operations  and  therefore  meets  the  criterion  of 
reconfigurability. 

Another  VE  demonstration,  the  Conning  Officer 
Virtual  Environment  (COVE)  project  at  the  Naval 
Air  Warfare  Center  Training  Systems  Division, 
was  designed  to  train  shiphandling  skills.  This 
immersive  VE  incorporates  a wide  variety  of  tools 
including  a head-mounted  display  (F1MD),  head 
tracking  devices,  voice  recognition,  cognitive 
modeling  of  spatial  knowledge,  and  various  task 
analyses. 

Future  systems  will  benefit  from  COVE  for  a 
number  of  reasons.  For  instance,  performance 
measures,  including  metrics  to  compare 
performance  of  trainees  using  F1MD  versus  using 
regular  desktop  displays,  will  be  developed. 
Those  types  of  analyses  will  help  clarify  the  added 
value  of  immersive  VE  versus  traditional  training 
methods.  Additionally,  to  the  extent  that  the 
technology  is  available,  the  COVE  project  is 
integrating  commercial  off-the-shelf  products, 
providing  a more  affordable  and  reconfigurable 
system  than  one  designed  with  custom-built 
components.  Finally,  the  COVE  project  will 
provide  a platform  for  other  vehicle  operations 
including  small  craft  and  amphibious  landing 
craft. 
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2.  System  description 

As  evident  in  the  VE  systems  above,  the  general 
approach  to  the  developing  VE  systems  is 
multidisciplinary.  The  designs  include  aspects  of 
psychology,  computer  modeling  and  simulation, 
user-interface  design,  and  robotics.  They  may 
also  include  sophisticated  hardware  such  as 
sensors,  fracking  devices,  HMDs,  and  large-scale 
projection  screens.  The  desired  complexity  of  the 
VE  will  determine  the  need  for  the  various 
components.  The  following  description  provides 
an  overview  of  some  of  the  key  components  of  a 
VE  as  seen  in  Figure  1. 

Environmental  data  capture 
Whether  the  device  is  used  for  real-time 
operations  or  for  mission  rehearsal,  the  data 
driving  these  systems  are  initially  captured  from 
the  external  (or  operational)  environment.  As 
situations  or  tasks  require,  environmental  data  can 
be  stored  and  used  to  generate  future  scenarios  for 
training,  mission  rehearsal,  or  debriefing 

purposes.  Regardless  of  the  intended  use,  the 
systems  should  accurately  represent  the 

operational  environment  (or  be  able  to  quickly 
generate  simulations  of  the  external  environment). 
The  level  of  fidelity  that  the  system  can  present 
will  depend  upon  the  capacity  of  the  sensors,  the 
extraction  of  relevant  information,  and  the  method 
of  data  presentation. 

System  requirements 

• Sensors  - Systems  rely  on  sensors  to  detect 
and  capture  data  from  the  external  world  (e.g., 
radar,  sonar,  etc.).  A misrepresentation  of  the 
external  environment  can  lead  to  the 
development  of  a faulty  mental  model  for  the 
user.  Subsequently,  conclusions  and 
decisions  based  on  faulty  information  may 
lead  to  actions  with  catastrophic 
consequences.  Therefore,  the  sensors  must 
accurately  capture  and  represent  the  details  of 
the  operational  environment. 

• Transcription  - The  system  must  have  the 
capability  to  extract  relevant  information  and 
discard  irrelevant  information  (“noise”) 
captured  from  the  external  environment. 
There  should  be  built  in  capabilities  to  modify 
the  level  of  granularity  at  which  the  data  are 
presented,  in  some  situations  providing  fine 


detail  may  be  distracting,  whereas  an  overall 
picture  may  be  more  useful.  Those  issues 
need  to  be  addressed  through  task  analyses 
and  comparison  studies. 

• Presentation  - After  the  data  are  captured  and 
transcribed  appropriately,  they  must  be 
presented  to  the  user  so  that  the  information  is 
useful.  Human  factor  analyses  can  help 
determine  how  to  exploit  human  attention  and 
present  infoimation  effectively. 

The  various  modes  of  presentation  in  VEs  can 
include  any  combination  of  the  following: 
Visual  - The  arrangement  of  visual 
displays,  the  content  of  the  display,  and 
the  ease  in  which  the  user  can  access 
specific  information  can  greatly  affect 
how  quickly  and  effectively  the  user 
comes  to  a decision.  Weeding  through 
irrelevant  information  can  slow  the 
decision  making  process  and  lead  to  more 
stressful  situations,  which  in  turn  can 
increase  the  room  for  error. 

Audio  - Studies  will  need  to  be  conducted 
to  determine  in  what  cases  sound  enables 
a decision  instead  of  or  in  addition  to 
vision.  Such  issues  are  important  to 
address  because  spatial  audio,  virtual 
sound,  and  the  integration  of  virtual  with 
real  sound  may  be  among  the  future 
components  of  VEs.  A few  of  the  areas 
researchers  will  need  to  examine  include 
whether  sound  in  VEs  enhances 
situational  awareness,  increases  the 
accuracy  of  localizing  objects,  or  aids  in 
judging  location  relative  to  sound  sources 
(i.e.,  to  aid  in  navigation). 

Haptics  - Haptics  refer  to  the  sense  of 
touch  derived  from  contact  forces.  Haptic 
interfaces  must  convey  a sense  of  touch  to 
a user  exploring  the  environment  to  create 
a feeling  of  immersion.  Haptics  can  aid 
in  tasks  such  as  locating  and  operating 
manual  controls  as  well  as  providing 
pilots  with  haptic  input  for  signaling  their 
direction  or  location. 

The  integration  of  visually,  aurally,  and 
haptically  presented  data  will  create  more 


28-3 


realistic  simulations  for  fully  immersive 
VEs  for  training  and  other  tasks.  The 
various  applications  for  multimodal, 
immersive  VEs  will  be  described  below 
(see  Section  3). 

The  human  operator 

These  systems  must  be  designed  with  the  user  in 
mind  because  certain  limitations  of  human 
sensory  processes  will  place  restrictions  on  the 
capacity  of  the  user  to  obtain  useful  information 
from  the  VE.  The  system  design  should  also  take 
into  account  the  most  effective  user  response.  For 
example,  will  a verbal  command  be  a more 
effective  response  than  a keystroke  or  push  of  a 
button? 

Feedback 

Whatever  the  result  of  the  user’s  action,  the 
system  should  have  the  capability  to  provide 
feedback  about  the  result.  Feedback  will  enable 
the  user  to  evaluate  and  learn  from  the  exercise, 
whether  it  is  real  or  training. 

Networked  systems 

The  above  description  applies  to  a single  (VE) 
unit,  however,  multiple  units  will  ultimately 
become  networked  over  systems  like  local  area 
networks  or  the  internet.  Networking  VE  systems 
will  have  an  great  advantage  over  traditional 
training  systems  in  that  they  can  be  remotely 
operated,  reach  a greater  number  of  trainees,  and 
enable  trainees  to  interact  across  multiple  sites. 

3.  Applications 

Military  Training  and  Operational  Tasks  - VE 
systems  will  continue  to  be  investigated  for  tasks 
such  as  expeditionary  warfare  in  urban  settings, 
mission  critical  duties,  various  types  of 
reconnaissance  (ground  and  air),  artillery  and 
surface  fire  support,  and  close  air  support  (CAS). 
As  mentioned  previously,  VE  trainers  will  be 
applied  to  the  piloting  of  ships,  small  craft,  and 
amphibious  landing  craft.  Individual  VEs  in  the 
form  of  wearable,  portable,  and  wireless  devices 
have  potential  to  facilitate  maintenance  tasks  and 
tactical  decision  support. 

Information  centers  - VEs  and  information 
displays  (e.g.,  command  information  centers)  will 
be  combined  to  create  multimodal, 
multifunctional  workstations.  As  evidence  of 


designs  for  highly  effective  displays  becomes 
available,  the  design  can  be  incorporated  into  VEs 
and  vice  versa.  To  further  such  a joint  venture,  a 
decision  support  program  sponsored  by  ONR  will 
study  through  task  analyses  whether,  for  example, 
3D  versus  2D  displays  are  more  effective.  The 
program  will  also  use  task  analyses  to  determine 
what  cognitive  processes,  what  spatial 
arrangements,  and  what  aspects  of  data  are  critical 
to  decision  support. 

Commercial  - The  entertainment  industry  already 
uses  VEs  for  a variety  of  functions,  such  as 
immersive  video  games.  VEs  are  also 
implemented  in  telecommunications,  information 
visualization,  product  design  and  manufacturing. 
Similarly,  large-scale  manufacturers  can  utilize 
the  technology  to  prototype  airplanes,  ships,  and 
other  vehicles  and  test  them  prior  to  costly 
investment  in  the  development  of  the  actual 
product. 

Medicine  - VEs  have  a number  of  applications  for 
medicine  from  planing  neurosurgery1  to  using 
virtual  patients  to  teach  medical  students  how  to 
deal  with  traumatic  injuries2  to  teaching  anatomy. 
Additionally,  simulators  have  been  created  to 
deepen  doctors’  understanding  of  cancer-related 
fatigue’.  Finally,  modeling  pharmaceutical 
compounds  can  aid  in  understanding  their 
mechanism  of  action  and  facilitate  the 
development  of  effective  treatments. 

Telecommunications  - Items  such  as  personal 
navigation  devices  or  multimodal,  wireless  phones 
(3rd  generation)  will  benefit  from  advancements  in 
VE  technology.  Similarly,  VE  technology  will 
benefit  from  advancement  in  wireless 
communications  for  networking  VEs  over 
multiple  sites. 


1 Kockro,  R.  A.,  et  al.  (2000).  Planning  and  Simulation  of 
Neurosurgery  in  a Virtual  Reality  Environment.  Neurosurgery,  46 
(1),  118-137. 

2 Lurie,  S.  (2000).  Innovation  and  service  traditional  at  University 
of  Michigan  Medical  School.  JAMA,  283  (7),  865-866. 

3 Vastag,  B.  & Beidler,  N.  (1998).  Tired  out:  patients  find  few  easy 
answers  for  cancer-related  fatigue.  Journal  of  the  National  Cancer 
Institute,  90,  1591-1594. 
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4.  Challenges 

The  promoters  of  VE  technology  have  made 
several  promises  about  the  utility  and  importance 
of  VEs.  Before  the  developers  can  deliver  on 
those  promises,  several  technical  and  scientific 
issues  must  be  resolved  - some  of  which  are 
outlined  and  described  below. 

• Visual  - current  visual  displays  are  limited  in 
terms  of  their  physical  and  geometric  fields- 
of-view,  depth  of  focus,  visual  contrast, 
resolution,  frame  rate,  weight,  and  display 
latency 

• Representation  of  self  (avatar)  - methods  to 
generate  egocentric  (a  representation  as  one 
would  normally  view  their  body)  or 
exocentric  views  need  to  be  developed 

• Direct  interaction  with  objects  (collision, 
control,  manipulation)  - haptic  interfaces 
need  to  be  improved 

• Sound  - methods  to  generate  spatial  audio  and 
virtual  sound  without  enormous  expense  need 
further  development 

• Multidisciplinary  approach  requires 
compatibility  among  system  components 
(computer  graphics,  behavioral  modeling, 
multimodal  interfaces,  etc.) 

• Faster  processors  for  real-time  display  of 
information  and  multimodal  synchronization 
are  needed 

• More  reliable  and  higher  speed  connections 
and  multi-user  interfaces  are  necessary  for 
networking 

In  the  context  of  training,  VE  systems  must 
demonstrate  efficacy  in  improving  task 

performance.  Further  research  is  needed  to 
determine  the  extent  that  3D,  immersive 
environments  create  greater  situational  awareness 
versus  traditional  methods.  Effective  metrics 

must  be  in  place  to  evaluate  the  utility  and 
efficacy  of  VEs  as  a teaching  tool.  Although  the 
development  of  standard  measures  will  require  a 
concerted  effort  on  the  part  of  program  managers 
and  researchers,  it  is  nonetheless  achievable. 


5.  Summary 

VEs  are  dynamic,  complex,  and  powerful  tools  for 
information  presentation.  The  capacity  of  VEs  to 
supply  multidimensional  representations  of  data 
can  dramatically  change  the  manner  in  which 
those  data  are  examined.  The  broad  spectrum  of 
application  domains  for  VE  technology  has 
naturally  generated  interest  from  a number  of 
disciplines.  As  such,  one  of  the  greatest 
challenges  to  advancing  VE  technology  as  whole 
is  creating  a collaborative  environment  where 
disparate  fields  can  mutually  benefit  from  one 
another  and  advance  VE  technology  to  its  fullest 
potential. 
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Figure  1 . A framework  for  VE  systems.  See  text  for  details. 
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